Method and System to Detect Failure in PCIe Endpoint Devices

ABSTRACT

A method, system, apparatus, and architecture are provided for detecting failure of a PCIe endpoint device by scanning an extended configuration space for each connected PCIe endpoint device to detect a first PCIe endpoint device that supports advance status reporting, and then by programming a first predetermined value and a second predetermined value, respectively, into an endpoint response register and a root complex request register of a dedicated memory control word in the extended configuration space for the first PCIe endpoint device, where the second predetermined value signals a request to the first PCIe endpoint device to update the endpoint response register of the dedicated memory control word with a new status value so that, after a minimum specified delay, a report that the first PCIe endpoint device has failed may be generated in response to detecting that the first predetermined value is stored in the endpoint response register.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is directed in general to high-speed serial busesand associated methods of operation. In one aspect, the presentinvention relates to endpoint device failure detection on a high-speedperipheral component interconnect express bus.

Description of the Related Art

PCI-Express (PCIe) is the backbone of today's complex systems requiringhigh speed data communication with high throughput, and is useful forenhancing a limited functionality system (e.g., computer processing unitor CPU) by specifying a very high-speed data communication link that canbe connected to a system having the missing functionality (e.g., agraphics card). Used extensively in different applications, such ascomputer cards, graphic cards, automotive networks, networking,industrial and consumer applications, PCIe is useful for very high-speeddata applications for real-time graphics and video processing, such asrequired by Advanced Driver Assistance Systems (ADAS). The PCIespecification includes a set of advanced error reporting and errorlogging features to detect and specify the type of error that occurs ina PCIe link, such as receiver errors, Data Link Layer Packet CyclicalRedundancy Checking (CRC) errors, Transaction Layer Packet (TLP) CRCerrors, and malformed TLP errors. To assist with error detection and/orcorrection by the host CPU, the PCIe Specification provides that eachPCIe endpoint device includes a Link Training and Status State Machine(LTSSM) hardware which detects certain lane or communication linkfailures and enters a Recovery state which sends an interrupt to thehost/CPU via the root complex (RC) so that the faulty link is eliminatedor reset by the host CPU. However, there are some PCIe endpoint devicefailures that are not detected by the LTSSM hardware, such as when thePCIe endpoint device is located behind a switch, in which case the LTSSMstate will not reach the software running on the host CPU. In addition,when the failure is due to a software state of the device, such as whenthe CPU crashes or is stuck in a while loop, etc., the LTSSM state willnot reach the host CPU. As seen from the foregoing, the existing PCIeerror detection solutions are extremely difficult at a practical levelby virtue of the challenges with providing flexible and comprehensiveerror detection and correction hardware within the performance, design,complexity and cost constraints of existing PCIe interconnect systems.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be understood, and its numerous objects,features and advantages obtained, when the following detaileddescription of a preferred embodiment is considered in conjunction withthe following drawings.

FIG. 1 is a block diagram of a PCIe communication system which providesendpoint device failure advance reporting in accordance with selectedembodiments of the present disclosure.

FIG. 2 is a simplified block diagram illustration of the hardware andsoftware components of the endpoint device failure advance reportingsystem in accordance with selected embodiments of the presentdisclosure.

FIG. 3 shows PCIe device configuration space registers used to implementendpoint device failure advance reporting in accordance with selectedembodiments of the present disclosure.

FIG. 4 depicts a simplified flow chart showing the logic for a hostsoftware to initiate an advance status reporting procedure in accordancewith selected embodiments of the present disclosure.

FIG. 5 depicts a simplified flow chart showing the logic for a hostsoftware to periodically program and read advance status reportingregisters in the PCIe configuration space of each PCIe endpoint devicein accordance with selected embodiments of the present disclosure.

DETAILED DESCRIPTION

An advance status reporting method, device, system, and program code aredescribed for discovering an endpoint device failure on a PCI Expressinterconnect bus by defining a dedicated memory word (4 bytes) in theextended configuration space for each endpoint device which isperiodically set by host software to first value for soliciting endpointdevice status reporting and then monitored by host software to detectany failure by the endpoint device to provide a status report, therebyindicating a failure in the endpoint device(s). In selected embodiments,advance status reporting is enabled by using a capabilities pointer inthe extended PCI configuration space of a PCIe endpoint device orswitch, or PCIe bridge to point to the next capabilities pointer in thePCIe capabilities register set which, in turn, points to the dedicatedmemory word which includes a first byte (Byte 0) which contains anadvance reporting capability indication for the EP device, a second byte(Byte 1) which contains the next capabilities pointer, a third byte(Byte 2) which contains a root complex request, and a fourth byte (Byte3) which contains the endpoint response received from the EP device. Inoperation, host software is configured during enumeration to check eachEP device's first byte (Byte 0) to determine if the device supportsadvance reporting and to start a thread for advance status reportingfrom each EP device which supports advance reporting. In addition, thehost software may be configured to periodically clear the fourth byte(Byte 3) to a 0 value and to program the third byte (Byte 2) to requestadvance status reporting in the EP device configuration space, therebytriggering the EP device to collect the required status and update thefourth byte (Byte 3) with the EP response. By periodically checking tosee if the fourth byte (Byte 3) contains a 0 value, the host softwarecan report the EP device failure to upper software to initiate devicerecovery.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 1which depicts a block diagram of a PCIe communication system 100 whichprovides a CPU 101 and memory 103 that are communicatively coupledthrough a root complex 106 to access any of the PCIe devices 120, 130,140, 150, 160, 170 using the PCIe switch-based serial interconnect. Asused herein, PCIe (peripheral component interconnect express) is aninterface standard for connecting high-speed peripheral devices orcomponents to a computer (also referred to as a central processing unit(CPU), a computing device, a server, a data processing system, etc.)based on a point-to-point topology. With the depicted arrangement, theroot complex 106 can directly access the memory 103 over the bus 104without CPU intervention, just like DMA. In addition, the CPU 101 usesthe root complex 106 to access any of the PCIe devices 120, 130, 140,150, 160, 170 and to generate interrupts to the CPU 101 for any of theevents generated by root complex 106 or the events generated by anyother PCIe devices 120, 130, 140, 150, 160, 170.

In the illustrated example, the root complex 106 may include a firstroot port 112A for connection over the PCIe bus 112 to a switch port112B of a switch 120, where the first root port 112A, PCIe bus 112, andswitch port 112B are collectively identified as the port/bus link112A/B. In turn, the switch 120 is connected, respectively, over PCIeport/bus links 115A/B, 116A/B, 117A/B to one or more PCIe endpointdevices 130, 140, 150. In addition, the root complex 106 may include asecond root port 113A for direct connection over a PCIe bus 113 to anupstream port 113B of the PCIe endpoint device 160. The root complex 106may also include a third root port 114A that is directly connected overa PCIe bus 114 to an upstream port 114B of the PCIe Bridge 170 toPCI/PCI-X bus 180 that provides a connection between a PCIe link and aPCI/PCI-X link. The PCIe buses 112-117 are logical connections (a.k.a.,interconnects or links) which are used to connect the PCIe endpointdevices 120, 130, 140, 150, 160, 170 and to detect the status andbandwidth of each PCIe endpoint device (e.g., PCIe endpoint device 130).As used herein, a link is a point-to-point communication channel betweentwo PCIe ports that allows both ports to send and receive PCIe requests(e.g., configuration, IO or memory read/write) and interrupts (e.g.,INTx, MSI or MSI-X). At the physical level, a link is composed of one ormore lanes. For example, a low-speed peripheral device (e.g., an 802.11Wi-Fi card) uses a single-lane (xl) link, while a graphics adaptertypically uses a much wider and faster 16-lane link.

In the depicted PCIe communication system 100, the CPU 101 may be anysuitable processing device, such as a processor, that may be programmedwith an software instructions and/or programming code. Thus, the CPU 101may be embodied with one or more processors, each having one or moreprocessor cores included therein. The CPU 101 may represent one or moregeneral-purpose processors such as a microprocessor, a microcontroller,or the like. For example, the CPU 101 may be a complex instruction setcomputing (CISC) microprocessor, reduced instruction set computing(RISC) microprocessor, very long instruction word (VLIW) microprocessor,or processor implementing other instruction sets, or processorsimplementing a combination of instruction sets. The CPU 101 may also beone or more special-purpose processors such as an application specificintegrated circuit (ASIC), a cellular or baseband processor, a fieldprogrammable gate array (FPGA), a digital signal processor (DSP), anetwork processor, a graphics processor, a network processor, acommunications processor, a cryptographic processor, a co-processor, anembedded processor, or any other type of logic capable of processinginstructions.

Connected to the CPU 101 over a memory bus 105 is a memory 103 which maystore programming code and/or software instructions for execution by theCPU 101. The memory 103 may include one or more volatile storage (ormemory) devices such as random access memory (RAM), dynamic RAM (DRAM),synchronous DRAM (SDRAM), static RAM (SRAM), or other types of storagedevices. In addition or in the alternative, the memory 103 may includenon-volatile memory, such as read only memory (ROM), electricallyerasable programmable ROM, flash memory, or the like. In whatever form,the memory 103 may store information including sequences of instructionsthat are executed by the CPU 101 or any other device. For example,executable code and/or data, in including but not limited to anoperating system 10, device drivers 20, firmware (e.g., input outputbasic system or BIOS), and/or applications can be loaded in the memory103 and executed by the processor 101. In addition and as describedhereinbelow, the memory 103 may also store an endpoint device failureadvance reporting module 30.

As will be appreciated in the context of the present disclosure, theoperating system 10 may include system software that manages andcontrols the resources in the PCIe communication system 100, includingbut not limited to basic input/output system (BIOS) programming and/orwith host software for implementing specified PCIe functionality, suchas, for example, processing interrupt requests and/or recovery loops toperform an endpoint device enumeration, restart, link restart, and/orbus rescan. For example, operating system 10 may include one or moreutility tools configured to discover, test, retrain and/or repair one ormore PCIe devices 120, 130, 140, 150, 160, 170 and their respective rootcomplexes (e.g., 106), and to access their PCIe configuration spaceheaders in order to read the statuses of the discovered PCIe devices andmodify the states of the discovered PCIe devices. In selectedembodiments, the operation system 10 may include a device discoverymodule, testing module, and link repair/retrain module, where eachmodule may be implemented in software, hardware, or a combinationthereof. In selected embodiments, device discovery module may be used todetect one or more PCIe devices using a PCIe enumeration process whereinthe host system reads the configuration space of each PCIe endpointdevice. In addition, the operating system 10 may perform basic tasks,such as controlling and allocating virtual memory, prioritizing theprocessing of instructions, controlling virtualized input and outputdevices (e.g., PCIe devices), facilitating networking, managing avirtualized file system, etc. Any suitable operating system 10 can beused, including but not limited to Windows® operating system fromMicrosoft®, Mac OS®/iOS® from Apple®, Android® from Google®, Linux®,Unix®, or other real-time or embedded operating systems.

In keeping with the functionality of the operating system 10, the memory103 may also store one or more device drivers 20 that are used tointeract with the one or more PCIe devices 120, 130, 140, 150, 160, 170.Device drivers 20 may be a computer software that provides interactionwith a hardware device (e.g., PCIe device 130) to an OS (e.g., operatingsystem 10) or other application software. For example, due to thespecialized role of a device driver, device drivers are oftenhardware-dependent and operating system-dependent, which may be runningwith a kernel of the operating system.

The PCIe communication system 100 also includes a root complex 106 thatis connected to the CPU 101 and memory 103. Located at the uppermostlayer of the tree structure of the PCIe communication system 100, theroot complex 106 is respectively connected through a system or frontside bus 102 with CPU 101 and through a memory bus 104 with memory 103.The root complex 106 includes IP registers 108 which are used toconfigure the IP. In particular, the IP registers 108 are used to enablethe clocks, to program the files, to configure the lane width (1 lane or2 lane), and/or to configure the speed mode (gen 1 or gen2 or gen3). Inaddition, the root complex registers 108 are used by the CPU 101 todefine a configurable address space for translating addresses from CPUaddress space to the PCIe address space. In the PCIe system, there arefour spaces: memory space, I/O space, message space and configurationspace. Apart from message space, the other spaces will have physicaladdress associated with it. The size of the configuration space is 4 KBwhile that of an I/O space is 64 KB. Configuration space will have allthe information about the device. It has device ID, vendor ID, classcode and various capabilities of the device. It is a software backwardcompatible to a PCI which has a size of 256 Bytes. Of this 4 KB ofconfiguration space, the first 64 bytes are standard and are called asStandardized Headers. These Standardized Headers are of two types (Type0 & Type 1). Type 0 will be used by the PCIe end points (e.g., 130) andwill have information that is applicable to end points. Type 1 will beused by root ports (e.g., 112A), switches (e.g., 120), and bridges(e.g., 170). It contains information applicable only to those. EveryPCIe endpoint device 120, 130, 140, 150, 160, 170 will also have a PCIeconfiguration space (PCS), but with a different type of header from thatused in root complex 106.

Connected to the root complex 106 are a plurality of PCIe switches 120,endpoint devices 130, 140, 150, 160, and/or bridges 170. For example,the PCIe communication system 100 may include a switch 120 as atransaction layer packet (TLP) relay device between the root complex 106and PCIe endpoint device(s) 130, 140, 150. As will be appreciated, aswitch may implement a logical assembly of multiple virtual PCIe bridgedevices (i.e., logical PCIe-to-PCIe bridges), such as a network switchthat controls multiple point-to-point serial connections by using packetswitching to receive, process, and forward data to each endpoint device(e.g., PCIe endpoint devices 130, 140, 150). For example, switch 120 maybe configured to provide a fan-out from root complex 106 to links115A/B, 116A/B, 117A/B, and to provide link scaling so that theavailable PCIe bus bandwidth is allocated, such that a predeterminednumber of links 115A/B, 116A/B, 117A/B, each having a size conforming toPCIe architecture standards, are physically routed to PCIe endpointdevices 130, 140, 150. Under the PCIe specification, each link (e.g.,115A/B) includes one or more lanes, with a link having a single lane(also referred to as having a xl width) being implemented with twolow-voltage differential pairs to provide a dual simplex serialconnection between two PCIe devices. In addition, the depicted PCIecommunication system 100 may include a PCIe endpoint device 160 that isdirectly connected to the root complex 106 via PCIe bus 113A/B (e.g.,without having to use switch 120). For example, if the PCIecommunication system 100 is embodied as a desktop computer, the PCIeendpoint device 160 may be a graphics adapter, a hard disk drive (HDD)(via a serial ATA link), or the like. Alternatively, if the PCIecommunication system 100 is embodied as a server, the PCIe endpointdevice 160 may be a gigabit Ethernet (GbE) and/or an additional bridgedevice.

To support advance status reporting for discovering an endpoint devicefailure on a PCI Express interconnect bus, the root complex 106 mayinclude a centralized endpoint device failure advance reporting module110 which is communicatively coupled to one or more PCIe endpointdevices 130, 140, 150, 160, 170 which each include an advance statusreporting module 132, 142, 152, 162, 172. In addition, each PCIeendpoint device 120, 130, 140, 150, 160, 170 includes a PCI extendedconfiguration space (PCS) 121, 131, 141, 151, 161, 171 which includes adedicated memory word (4 bytes) in the extended PCIe configuration spacefor each endpoint device. In each PCIe endpoint device (e.g., 130), adedicated memory word (e.g., 4 bytes) is defined in the PCIeconfiguration space (e.g., 131) by the host CPU 101 for periodicallysoliciting endpoint device status reporting that can be monitored by thehost software to detect any failure by the endpoint device to provide astatus report, thereby indicating a failure in the endpoint device(s).

Though illustrated as being located in the root complex 106, thecentralized endpoint device failure advance reporting module 110 may beexecuted in whole or in part by the host CPU 101 as host software in theform of the EP device failure advance reporting module 30. Alone or incombination with an advance status reporting module 132, 142, 152, 162,172 at each PCIe endpoint device, the host software sets up the extendedPCI configuration space (e.g., 131) of each PCIe endpoint device orswitch or bridge (e.g., 130) to include a capabilities pointer whichpoints to the next capabilities pointer in the PCIe capabilitiesregister set which, in turn, points to the dedicated memory word. In anexample embodiment, the dedicated memory word in the PCIe configurationspace of each PCIe endpoint device includes a first advance reportingcapability indication byte (Byte 0) for the EP device, a second nextcapabilities pointer byte (Byte 1), a third root complex request byte(Byte 2), and a fourth endpoint response byte (Byte 3) which containsany response received from the EP device. Of course, the order andsequence of bytes in the dedicated memory word can be changed. Inoperation, host software is configured during enumeration to check eachEP device's advance reporting capability indication byte (e.g., Byte 0)to determine if the EP device supports advance reporting and to start athread for advance status reporting from each EP device which supportsadvance reporting. In addition, the host software may be configured toperiodically clear the endpoint response byte (e.g., Byte 3) to a 0value and to program the root complex request byte (e.g., Byte 2) torequest advance status reporting in the EP device configuration space,thereby triggering the EP device to collect the required status andupdate the fourth byte (Byte 3) with the EP response. By periodicallychecking to see if the endpoint response byte (e.g., Byte 3) contains a0 value, the host software can report the EP device failure to uppersoftware to initiate device recovery.

As will be appreciated, the PCIe communication system 100 may beimplemented with a data processing system, such as a personal computersystem, server, workstation, network storage device, embedded system, orany other suitable computer system. Generally speaking, a “dataprocessing system” may include any instrumentality or aggregate ofinstrumentalities operable to compute, classify, process, transmit,receive, retrieve, originate, switch, store, display, manifest, detect,record, reproduce, handle, or utilize any form of information,intelligence, or data for business, scientific, control, or otherpurposes. As such, the data processing system may include random accessmemory (RAM), one or more processing resources (such as a CPU, hardwareor software control logic), read only memory (ROM), and/or other typesof memory. Additional components of the data processing system mayinclude one or more disk drives, one or more network ports forcommunicating with external devices, as well as various input and output(I/O) devices, such as a keyboard, a mouse, and a video display. Thedata processing system may also include one or more buses operable totransmit communications between the various hardware components.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 2which depicts a simplified block diagram 200 of the hardware andsoftware components of an endpoint device failure advance reportingsystem which is embodied as a PCIe communication system. As indicatedwith the hardware legend 221, the hardware system components include ahost CPU 207 which is connected over a front side bus 208 to the rootcomplex hardware 209. In addition, the root complex hardware 209 isconnected over a first PCIe bus 210 to a first endpoint device 211, andis also connected over a second PCIe bus 214 to a switch 215. In turn,the hardware switch 215 is connected over a third PCIe bus 217 to asecond endpoint device 219, and is also connected over a fourth PCIe bus218 to a third endpoint device 222.

As indicated with the software legend 222, the software systemcomponents include a host driver 201 and endpoint driver 203 which arestored in memory and executed by the host CPU 207 to exchangeinformation 202 with one another. In addition, the host CPU 207 andsoftware drivers 201, 203 are communicatively coupled to exchange dataand/or instructions 205, 206 and to control the endpoint devices (e.g.,211) with configuration data 204. In this arrangement, the host CPU 207and software drivers 201, 203 are communicatively coupled through theroot complex 209 to access the PCI configuration space 212 in the firstPCIe endpoint device 211 using the PCIe switch-based serial interconnectbusses 208, 210. In similar fashion, the host CPU 207 and softwaredrivers 201, 203 are communicatively coupled through the root complex209 to access the PCI configuration space 216 in the switch 215, as wellas the PCI configuration spaces 220, 223, respectively, in the PCIeendpoint devices 219, 222 using the PCIe switch-based serialinterconnect busses 208, 214, 217, 218. In addition, the CPU 207 may beoperatively coupled through the root complex 209 and/or device drivers201, 203 to access the advance status reporting modules 213, 221, 224 inthe PCIe endpoint devices 211, 219, 222 to perform advance reporting ofthe endpoint device status to the root complex 209 and/or host CPU 207.

As disclosed herein, each of the advance status reporting modules 213,221, 224 may be configured to specify, at each endpoint device, areporting word having a set of specified values or bytes which enablethe host CPU 207 to detect when an endpoint device has failed and thenundertake necessary action to restart the endpoint device. To this end,each reporting word stored in the PCIe configuration space (e.g., 212)of an endpoint device (e.g., 211) may include a capability byte or valuewhich signals whether the endpoint device has an advance reportingcapability. In addition, the reporting word may include an endpointresponse status byte or value which is periodically cleared by the hostCPU 207 in preparation of receiving a status report from the endpointdevice that is functional. In addition, the reporting word may includean endpoint status request byte or value which is periodically set bythe host CPU 217 to trigger or prompt the endpoint device to provide astatus report. By clearing the endpoint response status byte/value andsetting the endpoint status request byte/value, the host CPU 207 cansubsequently check the endpoint response status byte/value to see if theendpoint device wrote a response to the endpoint response statusbyte/value. If so, then the endpoint device is functional. If not, thenthe endpoint device is not functional, in which case the host CPU 207software may invoke a recovery loop and/or perform a device restart orlink restart and issue rescan of the bus (e.g. 210).

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 3which shows PCIe device configuration space registers 300 which may beused to implement endpoint device failure advance reporting. Asillustrated, the PCIe device configuration space registers 300 includesa PCIe configuration space header 301 having a shared set of commonregister fields. These registers are defined for both Type 0 and Type 1configuration space headers. As illustrated, the PCIe configurationspace header 301 includes, inter alia, a Device ID register, a Vendor IDregister, a Status register, a Command register, as well as one or moreBase Address Registers and an 8-bit capabilities pointer register 302,and so on. The Device ID register identifies a unique ID to describe thePCIe device itself. The Vendor ID register identifies a unique ID todescribe the originator of the PCIe device. Further, the Device ID (DID)and Vendor ID (VID) registers are both used to identify the PCIe device,and are commonly called the PCIe ID (or PCI ID). For example, the 16-bitvendor ID is allocated by the PCI-SIG, and the 16-bit device ID is thenassigned by the vendor.

The depicted PCIe configuration space header 301 includes a set of bitposition numbers at the top of the header table (e.g., 0, 15, 16, and31) that mark the bit position in the registers of PCIe configurationspace header 301. In addition, the depicted PCIe configuration spaceheader 301 includes a set of numbers on the right of header table (e.g.,00h-3Ch) that mark the byte offset of the registers in PCI configurationspace header 301. For example, to address a PCIe device, it must beenabled by being mapped into the system's 10 port address space ormemory-mapped address space. For example, the system's OS 10, devicedrivers 20, or firmware may be configured to program the Base AddressRegisters (e.g., BAR0, BAR1) to identify the address mapping of a PCIedevice. As such, processing logic of PCIe communication system 100 maydiscover one or more PCIe devices under test (DUT) and the rootcomplexes of the respective PCIe devices to identify a physical PCIeslot in which a PCIe device is inserted/reinserted, such that the one ormore PCIe devices may be concurrently identified and tested. Note thateach PCIe device includes capabilities pointer register 302 whichprovides a mechanism to extend the configuration space of a PCIe device(as compared to a PCI device) by specifying a pointer 303 to a PCIeconfiguration space extension 304 (a.k.a., PCI capabilities registerset). As depicted, the PCIe configuration space extension 304 mayinclude a PCIe Capabilities Register, a Next Capabilities Pointer 306, aPCIe Capabilities ID 305, a Device Capabilities Register, a DeviceStatus field, a Device Control field, a Link Capabilities register, aLink Status field, a Link Control field, a Slot Capabilities register, aSlot Status field, a Slot Control field, a RsvdP field, a Root Controlfield, a Root Status field, etc. For example, when the PCI capabilitiesregister set 304 is enabled, it is tied together by a linked list 303that starts with an 8-bit capabilities pointer 302 at address 34 h inPCIe device configuration space header 301. Note that each capabilitystructure set also has a unique capability ID. In particular, thecapabilities pointer register 302 points to the PCIe Capabilities ID 305at the start of PCIe capabilities register set 304 by using an 8-bitoffset (in bytes) 303 at the start of PCIe capabilities register set303. This 8-bit offset is stored in capabilities pointer register 302.While the position of PCIe capabilities register set 304 isdevice-specific, it is placed in the first 256 bytes of the PCIe deviceconfiguration space 300 and located after the mandatory PCI header. As aresult, the processing logic of PCIe communication system 100 may beused to access multiple PCIe configuration space headers (e.g., PCIecapabilities register set 304 and PCIe configuration space header 300)in order to read the statuses of the PCIe and root complex devices,modify the states of the PCIe and root complex devices, and/or clear anyerrors of the PCIe and root complex devices.

In the position of PCIe capabilities register set 304, a nextcapabilities pointer register 306 points to the start of a dedicatedmemory control word 311 stored in the PCIe configuration space 310 byusing a multi-bit offset (in bytes). As disclosed, the dedicated memoryword 311 may include a first advance reporting capability indicationbyte 321 (e.g., Byte 0) to indicate if the endpoint device supportsadvance reporting, a second next capabilities pointer byte 322 (e.g.,Byte 1) to provide a pointer to the next capability, a third rootcomplex request byte 323 (e.g., Byte 2) to trigger a status report fromthe endpoint device, and a fourth endpoint response byte 324 (e.g., Byte3) which contains any response received from the endpoint device. As aresult, the processing logic of PCIe communication system 100 may usethe Next Capabilities Pointer 306 to access the first advance reportingcapability indication byte (e.g., Byte 0) of the dedicated memorycontrol word 311 to determine if the endpoint device supports advancereporting. If so, the processing logic starts a thread for advancestatus reporting from the endpoint device, and then may periodicallyclear the fourth endpoint response byte (e.g., Byte 3) to a 0 value andprogram the third root complex request byte (e.g., Byte 2) to requestadvance status reporting in the endpoint device configuration space,thereby triggering the EP device to collect the required status andupdate the fourth endpoint response byte with the endpoint response. Byperiodically checking to see if the endpoint response byte (e.g., Byte3) contains a 0 value, the processing logic of PCIe communication system100 can report the endpoint device failure to upper software to initiatedevice recovery.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 4which depicts a simplified flow chart 400 showing the logic for a hostsoftware to initiate an advance status reporting procedure. In anexample embodiment, the control logic and methodology shown in FIG. 4may be implemented as hardware and/or software on a host computingsystem, processor, or microcontroller unit that includes processor andmemory for storing programming control code for controlling theoperation of one or more PCIe endpoint devices connected to a rootcomplex device.

The process starts (step 401) when the PCIe system begins theenumeration process of detecting which devices are connected to thehost. As will be appreciated, the enumeration process may be implementedby the root complex or host sending configuration packets to assignunique bus, device and function numbers to the detected endpoint (EP)devices. Once the enumeration process identifies any connected endpointdevices, the host software scans them by reading vendor ID register(step 402) and then reading all the capabilities of each connectedendpoint device (step 403). In reading the EP device capabilities, thehost will scan each PCIe EP device's extended configuration space tocheck which EP devices support advance status reporting (step 404). Inparticular, the host may be directed by pointers in the PCIe deviceconfiguration space registers to the dedicated advance status reporting(ASR) memory control word in the extended configuration space to detectif a first advance reporting capability indication byte (e.g., Byte 0)has a predetermined value (e.g., 0x5A) which indicates that advancestatus reporting is supported by the EP device. If the first advancereporting capability indication byte does not store the predeterminedvalue, this indicates that advance status reporting is not supported(negative outcome to detection step 404), and the process then restartsanother round of scanning the connected EP devices (step 402). However,if the first advance reporting capability indication byte does store thepredetermined value, this indicates that advance status reporting issupported (affirmative outcome to detection step 404), and the host theninitiates a thread for advance status report monitoring of the EP device(step 405). With the thread, the host is able to periodically monitorthe EP device status in runtime as indicated by the connection Step A(406).

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIG. 5which depicts a simplified flow chart 500 showing the logic formonitoring the EP device status with a host software which periodicallyprograms and reads advance status reporting registers in the PCIeconfiguration space of each PCIe endpoint device. In an exampleembodiment, the control logic and methodology shown in FIG. 5 may beimplemented as hardware and/or software on the host computing system,processor, or microcontroller unit that includes processor and memoryfor storing programming control code for controlling the operation ofone or more PCIe endpoint devices connected to a root complex device.

Continuing from the process 400 at connection Step A (501), themonitoring process begins at step 502 when, for each EP device whichsupports advance status reporting, the host programs a first value intothe endpoint response register (e.g., Byte 3) of the dedicated ASRmemory control word for each EP device. In selected embodiments, thehost software programs a first value by clearing the endpoint responseregister of the dedicated ASR memory control word to a 0 value. At step503, the host software then programs, for each EP device which supportsadvance status reporting, the second root complex request register(e.g., Byte 2) of the dedicated ASR memory control word to apredetermined value which will trigger a status report from the endpointdevice. In response to the programmed value, the corresponding EP devicecollects the required status information and updates the fourth endpointresponse register (e.g., Byte 3) of the dedicated ASR memory controlword with the status information (step 504). After a predetermined delay(e.g., 1 ms), the host software reads the endpoint response registerfrom the EP device configuration space (step 505) in order to see if thefirst value (e.g., 0) is still stored in the endpoint response register.If not (negative outcome to detection step 506), this indicates that theEP device is operative since the triggered EP device updated theendpoint response register to a different read value from the firstvalue, and the process returns to step 502 to periodically restart themonitoring process. However, the endpoint response register still storesthe first value (e.g., 0) (affirmative outcome to detection step 506),this indicates that the EP device failed or is otherwise not operativesince the triggered EP device was not able to update the endpointresponse register, and the process proceeds to step 507 to where thehost software reports the failure of the EP device to upper softwarecontrol to initiate EP device recovery, such as with a link restartprocess and subsequent re-enumeration of the EP device. At step 508, theprocess ends or alternatively returns to the enumeration process andcontinued periodic monitoring of EP devices.

By now it should be appreciated that there has been provided acomputer-implemented method, architecture, circuit, and system fordetecting failure of a peripheral component interconnect express (PCIe)endpoint device. In the disclosed methodology, a host software detectsthat one or more PCIe endpoint devices are connected to a dataprocessing system. In addition, the host software scans an extendedconfiguration space for each connected PCIe endpoint device to detect afirst PCIe endpoint device that supports advance status reporting. Thehost software also programs a first predetermined value into an endpointresponse register of a dedicated memory control word in the extendedconfiguration space for the first PCIe endpoint device. Subsequently,the host software programs a second predetermined value into a rootcomplex request register of the dedicated memory control word in theextended configuration space for the first PCIe endpoint device, wherethe second predetermined value signals a request to the first PCIeendpoint device to update the endpoint response register of thededicated memory control word with a new status value. After a minimumspecified delay, the host software reads the endpoint response registerof the dedicated memory control word and reports that the first PCIeendpoint device has failed in response to detecting that the firstpredetermined value is stored in the endpoint response register. Inselected embodiments, the dedicated memory control word is embodied as amemory word in the extended configuration space for the first PCIeendpoint device. In addition, the memory word may, in selectedembodiments, include a first advance reporting capability indicationregister for the first PCIe endpoint device, a second root complexrequest register for triggering a response by the first PCIe endpointdevice, and a third endpoint response register which contains anytriggered response received from the first PCIe endpoint device. In suchembodiments, the host software scans the extended configuration space byscanning the first advance reporting capability indication register todetect that the first PCIe endpoint device supports advance statusreporting. In addition, the host software may program the firstpredetermined value by resetting the third endpoint response register toa 0 value. In addition, the host software may program the secondpredetermined value to signal a request to the first PCIe endpointdevice to set the endpoint response register of the dedicated memorycontrol word with a 1 value. In selected embodiments, the disclosedmethodology may also include periodically repeating the steps ofprogramming the first predetermined value into the endpoint responseregister, programming the second predetermined value into the rootcomplex request register, and reading the endpoint response register ofthe dedicated memory control word.

In another form, there is provided a device for detecting failure of aperipheral component interconnect express (PCIe) endpoint device. Thedisclosed device includes at least one processing device configured toexecute a host operating system (OS), a first memory coupled to theprocessing device to store a Unified Extensible Firmware Interface(UEFI) table, and a communication bus operatively coupled to theprocessing device and configured for data communication with an endpointdevice. The disclosed processing device is to perform a specifiedsequence of steps. In particular, the processing device is configured todetect that one or more PCIe endpoint devices are connected tocommunication bus. In addition, the processing device is configured toscan an extended configuration space for each connected PCIe endpointdevice to detect a first PCIe endpoint device that supports advancestatus reporting. In selected embodiments, each connected PCIe endpointdevice may include device firmware and memory registers that contain anextended configuration space for the connected PCIe endpoint device. Inother selected embodiments, the communication bus is a PeripheralComponent Interconnect Express (PCIe) serial expansion bus and theextended configuration space for each connected PCIe endpoint device isa PCIe extended configuration space. In addition, the processing deviceis configured to program a first predetermined value into an endpointresponse register of a dedicated memory control word in the extendedconfiguration space for the first PCIe endpoint device. In addition, theprocessing device is configured to program a second predetermined valueinto a root complex request register of the dedicated memory controlword in the extended configuration space for the first PCIe endpointdevice, where the second predetermined value signals a request to thefirst PCIe endpoint device to update the endpoint response register ofthe dedicated memory control word with a new status value. In selectedembodiments, the dedicated memory control word includes a first advancereporting capability indication register for the first PCIe endpointdevice, a second root complex request register for triggering a responseby the first PCIe endpoint device, and a third endpoint responseregister which contains any triggered response received from the firstPCIe endpoint device. In such embodiments, the processing device may beconfigured to scan the extended configuration space for each connectedPCIe endpoint device by scanning the first advance reporting capabilityindication register to detect that the first PCIe endpoint devicesupports advance status reporting. In addition or in the alternative,the processing device may be configured to program the firstpredetermined value by resetting the third endpoint response register toa 0 value. In addition or in the alternative, the processing device maybe configured to program the second predetermined value to signal arequest to the first PCIe endpoint device to set the endpoint responseregister of the dedicated memory control word with a 1 value. Inaddition, the processing device is configured to wait for a minimumspecified delay before reading the endpoint response register of thededicated memory control word. Finally, the processing device isconfigured to report that the first PCIe endpoint device has failed inresponse to detecting that the first predetermined value is stored inthe endpoint response register. In selected embodiments, the processingdevice may also be configured to periodically repeating the steps ofprogramming the first predetermined value into the endpoint responseregister, programming the second predetermined value into the rootcomplex request register, and reading the endpoint response register ofthe dedicated memory control word.

In yet another form, a processing system is provided which includeslogic instructions on a non-transitory storage medium. In selectedembodiments, the logic instructions may be implemented as a computerprogram product which is embodied as a non-transitory machine-readablemedium having instructions stored therein for execution by a processorto periodically test for failure of a peripheral component interconnectexpress (PCIe) endpoint device. As disclosed, the logic instructions mayembody host software that is executed by the processor detect that oneor more PCIe endpoint devices connected to a communication bus supportadvance status reporting. In addition, the logic instructions or hostsoftware may be configured to reset an endpoint response register of anextended configuration space for each of the one or more PCIe endpointdevices to a first predetermined value. In addition, the logicinstructions or host software may cause the processor to program apredetermined response trigger value in a root complex request registerof the extended configuration space for each of the one or more PCIeendpoint devices, where the predetermined response trigger value signalsa request to a corresponding PCIe endpoint device to update the endpointresponse register with a new status value. In selected embodiments, eachof the one or more PCIe endpoint devices may include an extendedconfiguration space wherein a dedicated memory control word is storedwhich comprises the endpoint response register and the root complexrequest register. In such embodiments, the dedicated memory control wordfor each of the one or more PCIe endpoint devices may also include anadvance reporting capability indication register for storing apredetermined advance status reporting value which indicates that thecorresponding PCIe endpoint device supports advance status reporting. Inaddition, the logic instructions or host software may cause theprocessor to wait until after a minimum specified delay before readingthe endpoint response register of each of the one or more PCIe endpointdevices to detect if said PCIe endpoint device has failed to respond tothe predetermined response trigger value. In selected embodiments, thelogic instructions may be configured to report that a first PCIeendpoint device has failed in response to detecting that the firstpredetermined value is stored in the endpoint response register for thefirst PCIe endpoint device. In other selected embodiments, the logicinstructions may be configured to report that a first PCIe endpointdevice has not failed in response to detecting that the firstpredetermined value is not stored in the endpoint response register forthe first PCIe endpoint device.

Although the described exemplary embodiments disclosed herein focus onexample PCIe interconnect devices, systems, and methods for using same,the present invention is not necessarily limited to the exampleembodiments illustrate herein. For example, various embodiments ofproviding advance status reporting for PCIe endpoint device failuredetection may be applied in any suitable data processing systemapplication, and may use additional or fewer circuit components thanthose specifically set forth. Thus, the particular embodiments disclosedabove are illustrative only and should not be taken as limitations uponthe present invention, as the invention may be modified and practiced indifferent but equivalent manners apparent to those skilled in the arthaving the benefit of the teachings herein. Accordingly, the foregoingdescription is not intended to limit the invention to the particularform set forth, but on the contrary, is intended to cover suchalternatives, modifications and equivalents as may be included withinthe spirit and scope of the invention as defined by the appended claimsso that those skilled in the art should understand that they can makevarious changes, substitutions and alterations without departing fromthe spirit and scope of the invention in its broadest form.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any element(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature or element of any or all the claims. As used herein, the terms“comprises,” “comprising,” or any other variation thereof, are intendedto cover a non-exclusive inclusion, such that a process, method,article, or apparatus that comprises a list of elements does not includeonly those elements but may include other elements not expressly listedor inherent to such process, method, article, or apparatus.

What is claimed is:
 1. A computer-implemented method for detecting failure of a peripheral component interconnect express (PCIe) endpoint device, comprising: detecting, by host software, that one or more PCIe endpoint devices are connected to a data processing system; scanning, by host software, an extended configuration space for each connected PCIe endpoint device to detect a first PCIe endpoint device that supports advance status reporting; programming, by host software, a first predetermined value into an endpoint response register of a dedicated memory control word in the extended configuration space for the first PCIe endpoint device; programming, by host software, a second predetermined value into a root complex request register of the dedicated memory control word in the extended configuration space for the first PCIe endpoint device, where the second predetermined value signals a request to the first PCIe endpoint device to update the endpoint response register of the dedicated memory control word with a new status value; after a minimum specified delay, reading, by host software, the endpoint response register of the dedicated memory control word; and reporting, by the host software, that the first PCIe endpoint device has failed in response to detecting that the first predetermined value is stored in the endpoint response register.
 2. The computer-implemented method of claim 1, where the dedicated memory control word comprises a memory word in the extended configuration space for the first PCIe endpoint device.
 3. The computer-implemented method of claim 2, where the memory word comprises a first advance reporting capability indication register for the first PCIe endpoint device, a second root complex request register for triggering a response by the first PCIe endpoint device, and a third endpoint response register which contains any triggered response received from the first PCIe endpoint device.
 4. The computer-implemented method of claim 3, where scanning the extended configuration space for each connected PCIe endpoint device comprises scanning the first advance reporting capability indication register to detect that the first PCIe endpoint device supports advance status reporting.
 5. The computer-implemented method of claim 3, where programming the first predetermined value comprises resetting the third endpoint response register to a 0 value.
 6. The computer-implemented method of claim 3, where the second predetermined value signals a request to the first PCIe endpoint device to set the endpoint response register of the dedicated memory control word with a 1 value.
 7. The computer-implemented method of claim 1, further comprising periodically repeating the steps of programming the first predetermined value into the endpoint response register, programming the second predetermined value into the root complex request register, and reading the endpoint response register of the dedicated memory control word.
 8. A device for detecting failure of a peripheral component interconnect express (PCIe) endpoint device comprising: at least one processing device configured to execute a host operating system (OS); a first memory coupled to the processing device; and a communication bus operatively coupled to the processing device and configured for data communication with an endpoint device; where the processing device is configured to perform the following steps: detecting that one or more PCIe endpoint devices are connected to communication bus; scanning an extended configuration space for each connected PCIe endpoint device to detect a first PCIe endpoint device that supports advance status reporting; programming a first predetermined value into an endpoint response register of a dedicated memory control word in the extended configuration space for the first PCIe endpoint device; programming a second predetermined value into a root complex request register of the dedicated memory control word in the extended configuration space for the first PCIe endpoint device, where the second predetermined value signals a request to the first PCIe endpoint device to update the endpoint response register of the dedicated memory control word with a new status value; after a minimum specified delay, reading the endpoint response register of the dedicated memory control word; and reporting that the first PCIe endpoint device has failed in response to detecting that the first predetermined value is stored in the endpoint response register.
 9. The device of claim 8, where each connected PCIe endpoint device comprises device firmware and memory registers that contain an extended configuration space for the connected PCIe endpoint device.
 10. The device of claim 8, where the communication bus is a Peripheral Component Interconnect Express (PCIe) serial expansion bus and where the extended configuration space for each connected PCIe endpoint device is a PCIe extended configuration space.
 11. The device of claim 8, where the dedicated memory control word comprises a first advance reporting capability indication register for the first PCIe endpoint device, a second root complex request register for triggering a response by the first PCIe endpoint device, and a third endpoint response register which contains any triggered response received from the first PCIe endpoint device.
 12. The device of claim 11, where the processing device is configured to scan the extended configuration space for each connected PCIe endpoint device by scanning the first advance reporting capability indication register to detect that the first PCIe endpoint device supports advance status reporting.
 13. The device of claim 11, where the processing device is configured to program the first predetermined value by resetting the third endpoint response register to a 0 value.
 14. The device of claim 11, where the processing device is configured to program the second predetermined value to signal a request to the first PCIe endpoint device to set the endpoint response register of the dedicated memory control word with a 1 value.
 15. The device of claim 11, where the processing device is further configured to periodically repeating the steps of programming the first predetermined value into the endpoint response register, programming the second predetermined value into the root complex request register, and reading the endpoint response register of the dedicated memory control word.
 16. A processing system, comprising: logic instructions on a non-transitory storage medium configured to: detect that one or more PCIe endpoint devices connected to a communication bus support advance status reporting; reset an endpoint response register of an extended configuration space for each of the one or more PCIe endpoint devices to a first predetermined value; program a predetermined response trigger value in a root complex request register of the extended configuration space for each of the one or more PCIe endpoint devices, where the predetermined response trigger value signals a request to a corresponding PCIe endpoint device to update the endpoint response register with a new status value; and after a minimum specified delay, read the endpoint response register of each of the one or more PCIe endpoint devices to detect if said PCIe endpoint device has failed to respond to the predetermined response trigger value.
 17. The processing system of claim 16, where the logic instructions are configured to report that a first PCIe endpoint device has failed in response to detecting that the first predetermined value is stored in the endpoint response register for the first PCIe endpoint device.
 18. The processing system of claim 16, where each of the one or more PCIe endpoint devices comprises an extended configuration space wherein a dedicated memory control word is stored which comprises the endpoint response register and the root complex request register.
 19. The processing system of claim 18, where the dedicated memory control word for each of the one or more PCIe endpoint devices further comprises an advance reporting capability indication register for storing a predetermined advance status reporting value which indicates that the corresponding PCIe endpoint device supports advance status reporting.
 20. The processing system of claim 16, where the logic instructions are configured to report that a first PCIe endpoint device has not failed in response to detecting that the first predetermined value is not stored in the endpoint response register for the first PCIe endpoint device. 